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Description 

Field of the Invention 

This invention relates generally to video frame transmission and display. More particularly, the present invention 
relates to method and apparatus for converting interlaced scan video frame representations into progressively-scanned 
video frame representations. 

Background of the Invention 

Currently, video is represented as sequences of frames in accordance with either the interlaced or the progres- 
sively-scanned format. Each frame comprises a matrix of pixels that represent the luminance or the chrominance of 
an image. 

In the interlaced scan format, an image is represented using a pair of frames. The first frame of the pair, for example, 
includes pixels which are located only on alternate rows of the frame matrix, called horizontal lines. For the second 
frame of the pair, pixels are located in that frame matrix only in the corresponding horizontal lines which were missing 
pixels in the first frame, such that the portions of the image not represented in the first frame are represented in the 
second frame. The first and second frames of a pair are scanned consecutively on a video display monitor at a rate of 
60 frames/sec for purposes of reconstructing the entire image on the display at the industry interlaced scan standard 
30 frames/sec display rate. 

The progressively-scanned format is a more recently developed video representation technique. An image is rep- 
resented in its entirety using only a single frame which includes pixels in all horizontal lines of the frame matrix. Such 
frames, therefore, can be progressively scanned on a display at the standardized progressive display rate of 60 frames/ 
sec. 

It is well known that higher quality image reproductions may be obtained by using the progressively-scanned format 
rather than the interlaced scan format, because interlaced displays may typically be susceptible to line crawl and 
interline flicker, whereas progressively-scanned displays typically will not be. As a result, the use of the progressively- 
scanned format for video encoding and in video equipment, such as, for example, cameras, broadcast station trans- 
mitters and high definition television (HDTV) desktop or workstation display monitors, is currently expanding. Never- 
theless, it is foreseen that the interlaced scan format will continue to be widely used, because of the higher costs 
associated with implementing the progressively-scanned format technology. 

Several techniques for converting, or deinterlacing, a sequence of frames which have been represented using the 
interlaced scan format have been developed to provide that images which were initially represented using the interlaced 
scan format may be displayed on the higher image quality, progressively-scanned format display equipment. These 
techniques typically utilize a variety of well known spatial, temporal, or spatio-temporal interpolation processing for 
estimating the values of missing pixels in an interlaced frame. One deinterlacing technique, described in detail in A. 
Nguyen and E. Dubois, "Spatio-Temporal Adaptive Interlaced to Progressive Conversion, 0 in Proc. Int. Workshop on 
HDTV, November 1 992, incorporated by reference herein, estimates missing pixel values by performing a combination 
of spatial and temporal interpolations. This technique and other known deinterlacing techniques, however, do not ac- 
curately compute estimated values for missing pixels in an interlaced frame with a sufficient level of accuracy, such 
as, for example, to compensate for temporal artifacts, in order to provide for high quality image reconstruction through 
progressive scanning of the deinterlaced sequence of the frames. 

Summary of the Invention 

The present invention provides a method and apparatus for converting, or deinterlacing, a sequence of video 
frames which are represented using the interlaced scan format to a sequence of video frames represented according 
to the progressively-scanned format. 

In accordance with one aspect of the present invention, a plurality of interpolations are performed for computing 
approximations of estimated missing pixel values and, ultimately, generating a deinterlaced frame. The interpolations 
are performed in a sequence such that interpolations which require a more accurate approximation of missing pixel 
values for performing the respective interpolation estimations are performed subsequent to interpolations which require 
less accurate approximations of missing pixel values for performing an interpolation estimation. Interpolations are 
performed in this sequence to minimize the error in estimating missing pixel values for the respective interpolation 
computation. As a result, estimations from interpolations which depend on an approximated value of a missing pixel 
are computed with less error and can adequately contribute to the approximation of missing pixel values in combination 
with other interpolation estimations. The interpolations are weighted according to the errors each one introduces for 
generating the approximations of missing pixel values for a deinterlaced frame. 
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In another aspect of the present invention, deinterlacing of an interlaced frame comprised of luminance pixels is 
performed using a weighted combination of estimated values obtained from spatial and temporal interpolations. The 
spatial interpolations include vertical and orientational interpolations, and the temporal interpolations include linear 
temporal and bidirectional motion compensation interpolations. The vertical and linear temporal interpolations are per- 
5 formed first and use, respectively, only known pixel values of the frame being deinterlaced and known pixel values in 
the interlaced frames which, in time, precede and follow the frame being deinterlaced. Then, the orientational interpo- 
lation is performed using known pixel values of the frame being deinterlaced, and approximated missing pixel values 
for a deinterlaced frame which are based on the results of the previously performed vertical and linear temporal inter- 
polations. Finally, the bidirectional motion compensation interpolations are performed using known pixel values and 
io approximated missing pixel values. In particular, a backward interpolation estimation is performed using known and 
missing pixel values from a deinterlaced frame which, in time, precedes the frame being deinterlaced and has been 
approximated based on linear temporal, vertical, orientational and motion compensation interpolations. Further, a for- 
ward interpolation estimation is performed using known and missing pixel values from a deinterlaced frame which, in 
time, follows the frame being deinterlaced and has been approximated based on vertical, linear temporal and orienta- 
ls tional interpolations. The forward and backward motion compensation interpolations are weighted respectively with 
each other, and then combined, in relationship to the vertical, linear temporal and orientational interpolations, for gen- 
erating a final approximation of a deinterlaced frame. 

Alternatively, an interlaced frame comprising chrominance pixels is deinterlaced using the same sequence of ver- 
tical, temporal, orientational and motion compensation interpolations described above for the luminance frame dein- 
20 terlacing, and suitably modified for chrominance interlacing. 

Additional features and advantages of the present invention will become readily apparent to those of ordinary skill 
in the art by reference to the following detailed description and accompanying drawings. 

Brief Description of Drawings 

25 

FIGS. 1 A and 1 B illustrate a sequence of frames that represent images which are represented according to the 
interlaced scan format. 

FIGS. 2A and 2B illustrate a sequence of frames that represent images which are represented according to the 
progressively-scanned format. 

30 FIG. 3 is a flow diagram of a method for deinterlacing a sequence of interlaced frames in accordance with the 

present invention. 

FIG. 4A illustrates an interlaced frame upon which vertical interpolation may be performed for estimating missing 
pixel values in accordance with the present invention. 

FIG. 4B illustrates an interlaced frame upon which orientational interpolation may be performed for estimating 
35 missing pixel values in accordance with the present invention. 

FIG. 4C illustrates frames which are utilized for performing a linear temporal interpolation for estimating missing 
pixel values in an interlaced frame in accordance with the present invention. 

FIG. 4D illustrates frames which are utilized for performing a motion compensated interpolation for estimating 
missing pixel values in an interlaced frame in accordance with the present invention. 
40 FIG. 5 illustrates a generalized block diagram of functional blocks of an apparatus which may suitably perform 

operations associated with converting video frames represented using the interlaced scan format to video frames 
represented using the progressively-scanned format in accordance with the present invention. 

Detailed Description 

45 

The present invention uses interpolation techniques for converting, or deinterlacing, video frames which are rep- 
resented using the interlaced scan format to video frames represented using the progressively-scanned format. As 
more fully discussed below, missing pixel values in a frame being deinterlaced are estimated using interpolations which 
are performed in a sequence that requires increasing accuracy in the approximations of missing pixel values for per- 
so forming the respective interpolations in the sequence, thereby minimizing error in the interpolation estimations and in 
the approximation of a deinterlaced frame. Furthermore, the approximations of missing pixel values are computed 
using weighted combinations of the interpolations. 

At this point, an explanation of the representation of video images as sequences of frames using matrices of pixels, 
in accordance with the interlaced scan and progressively-scanned formats, respectively, is provided to allow for a better 
55 understanding of the deinterlacing technique of the present invention. 

FIG. 1 A illustrates a sequence of frames over time which are represented according to the interlaced scan format. 
FIG. 2A illustrates a sequence of frames over time which are represented according to the progressively-scanned 
format. In general, for both the interlaced scan and progressively-scanned formats, a frame comprises a matrix of 
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pixels. Each pixel in the matrix typically corresponds to an eight bit data value which is representative of either the 
intensity or color of the light in a spatially corresponding location of the represented image. As conventional in the art, 
a frame matrix is defined in terms of vertical pixel locations, v, and horizontal pixel locations, h. For purposes of clarity 
and simplicity, the present inventive technique is initially explained with reference to the deinterlacing of a sequence 
of frames whose pixels represent the intensity or luminance of an image. It is to be understood that the frames illustrated 
in the FIGURES and referred to in the text below only represent a portion of an image, and that the techniques described 
below may be used for estimating all missing pixel values for purposes of entirely deinterlacing an interlaced frame in 
accordance with the present invention. 

Referring to FIG. 1 A, the interlaced scan format provides that each image which is recorded or captured by, for 
example, a camera, is represented as a pair of frames. Each frame in the pair includes pixels representative of one- 
half of the entire image which was captured, such that both frames of pixels are required for reconstructing the image. 
By way of example and as illustrated in FIG. 1A, a pair of frames representative of an image may suitably include 
frames l ODD and l EVEN . The l ODD Uame includes pixels which are located only in odd numbered horizontal lines of the 
matrix, such as in lines v-1 and while the /a/sv frame includes pixels which are located only in even numbered 
horizontal lines, such as in lines v-2, vand v+2. 

FIG. 1B illustrates the appearance of pixels on a display resulting from scanning pairs of interlaced frames, such 
as the frames l ODD and l EEN o\ FIG. 1 A, on the display. As shown in FIG. 1B, when a sequence of pairs of interlaced 
frames is scanned on the display, the pixel data appear on alternate vertical lines over time, in an interlaced manner. 
The frames are typically scanned at 60 frames/sec, such that the entire image may be reconstructed according to the 
industry interlaced scan standard display rate of 30 frames/sec. 

In the progressively-scanned format, as shown in FIG. 2A, each frame contains pixels in all horizontal lines, such 
that the pixels in a single frame represent an entire image which has been captured. FIG. 2B illustrates the appearance 
of pixels in horizontal lines of a display when a sequence of these frames is progressively-scanned on a display. 

In comparing FIGS. 2A and 2B with FIGS. 1 A and 1 B, the locations which are missing pixels in an interlaced frame 
matrix are evident. The present invention provides a novel technique for accurately computing estimated values for 
the missing pixels in an interlaced frame, such that a sequence of deinterlaced frames may be generated for purposes 
of progressively scanning and reconstructing a high quality image on a display. 

In accordance with the present invention, interpolations are performed in a sequence for estimating missing pixel 
values. The sequence requires that interpolations which depend least upon previous interpolation processing, for ex- 
ample, interpolations which rely exclusively on known pixel values for estimating a missing pixel value, are performed 
before interpolations whose estimation accuracy depends substantially on previous interpolation processing for ob- 
taining approximated values for missing pixels in the frame being deinterlaced. Interpolations which require approxi- 
mations of missing pixel values for their computations include, for example, motion compensation which requires motion 
vectors for an image to be determined such that an estimate of missing pixel values may be accurately computed by 
that interpolation. In addition, when an approximation of missing pixel values is performed using more than one inter- 
polation estimation, the approximation is performed using a weighted combination of the interpolations being relied 
upon in order to weigh more heavily in the approximation those interpolations having the least error associated with 
the computation. 

For example, in the performance of a motion compensation interpolation, known pixel values in interlaced frames 
and approximations of missing pixel values in frames which have been deinterlaced to various orders of accuracy may 
be utilized for determining the motion vectors associated with computing estimated missing pixel values. 

Various levels of approximations of missing pixel values, in other words, deinterlaced frame approximations, may 
be generated, depending on the number of similar types of interpolations performed in a sequence. For example, a 
linear temporal and vertical interpolation may be performed, and then weighted in combination together at one stage 
of the sequence, because both depend only upon known pixel values in an interlaced frame for performing the esti- 
mation of missing pixel values in a frame being deinterlaced. 

In one preferred embodiment of the present invention, spatial interpolations are performed in combination with 
temporal interpolations to estimate missing pixel values in a frame being deinterlaced in accordance with the following 
generalized deinterlacing equation: 

hh.v.tl\>kh,v.ti Z{1[>)/,{h.vft. (1) 

In equation [1 ], the coordinates (h, v,t) identify a pixel location in a frame matrix, lj refers to the results of linear temporal 
and motion compensation interpolations along the estimated motion trajectory, /* refers to the result of spatial inter- 
polation and > is used for applying appropriate weighting factors to the spatial and temporal interpolations, respectively 
In a preferred embodiment, consists of vertical and orientational interpolation results, and t^. comprises linear tem- 
poral and motion-compensated interpolation results. It is to be understood that other interpolations may be performed 
and combined for generating an approximation of missing pixel values and, ultimately, a deinterlaced frame, depending 
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on the level of accuracy required for estimating missing pixel values using a particular interpolation process. 

FIG. 3 shows a process 50 for deinterlacing an interlaced frame in accordance with the equation [1]. The advan- 
tages of the process 50 are highlighted through the deinterlacing of an exemplary interlaced frame, l ODDf which is 
similar in structure and pixel arrangement to the frame l ODD shown in the FIG. 1 A. The frame l ODDt is reproduced in 

5 FIGS. 4A and 4B for purposes of illustrating vertical and orientational interpolations performed in accordance with the 
present inventive technique. FIGS. AC and 4D show how deinterlaced frames, which, in time, immediately precede 
and follow the frame \ ODDt and are at various orders of approximation accuracy, may be used for deinterlacing \qddv 
Referring to FIG. 4A, the frame l 0 DD t is missing pixels in alternate horizontal lines v, v+2 and v-2. The process 50 
attempts to estimate values for the missing pixels in these lines through spatial and temporal interpolation techniques. 

10 By way of example, the process 50 is performed for estimating the value of the missing pixel at position (h,v,t) in the 
frame l ODDr For ease of reference, l(h,v,t) is defined as the intensity of the missing pixel, MR at location (h,v,t)o\ l ODDt . 
Of course, the values of other missing pixels in the line v and in the entire frame l ODDt may be estimated using the 
process 50, such that a completely deinterlaced representation of the frame l ODDt may be generated for use with the 
progressive scanning format. 

15 in the exemplary embodiment of the present invention, the process 50 first approximates the values of missing 

pixels from the results of vertical and linear temporal interpolations. Then, missing pixel values are estimated from an 
orientational interpolation. The orientational interpolation finds a local edge displacement scalar using missing pixel 
values from the first approximation of the deinterlaced frame which was obtained from the vertical and linear temporal 
interpolation estimations. Finally, a motion compensated interpolation is performed using backward and forward motion 

20 compensation interpolations by first identifying backward and forward motion vectors, respectively. The backward mo- 
tion vector is determined using known and missing pixel values in a frame which, in time, immediately precedes the 
frame being deinterlaced, where the preceding frame is a frame which has been completely deinterlaced in accordance 
with the process 50. The forward motion vector is determined using known and missing pixel values in a frame which, 
in time, immediately follows the frame being deinterlaced, where the following frame is a frame which has been dein- 

25 terlaced by the performance of vertical, linear temporal and orientation interpolation estimations in accordance with 
the process 50. 

FIG. 5 shows an exemplary embodiment of an apparatus 10 for deinterlacing an interlaced sequence of frames 
in accordance with the process 50. The apparatus 10 suitably comprises individual functional blocks. The functions 
these blocks represent may be provided through the use of either shared or dedicated hardware, including but not 

30 limited to hardware capable of executing software. In a preferred embodiment, the functional blocks in the apparatus 
10 comprise a vertical interpolator 12, a linear temporal interpolator 14, an orientational interpolator 16, a bidirectional 
motion compensation interpolator 18, all of which are connected to a standard processor 22. The processor 22 is 
connected to a conventional memory 24, such as a ROM. As explained below, the functional blocks in the apparatus 
10 perform selected arithmetic computations, and provide data representative of the results of these computations to 

35 the processor 22. The processor 22 conventionally stores such data, as well as other processed data results, in the 
memory 24, as required. It is to be understood that the operations performed by the functional blocks and the processor 
22 may be similarly performed using a single shared processor. Such a processor may comprise a standard digital 
signal processor and would include read only memory or other appropriate memory for storing software and performing 
the operations discussed below Further, the present inventive technique may be implemented using firmware or by 

40 discrete components implemented in an integrated circuit. Other suitable embodiments may be readily implemented 
by those of ordinary skill in the art. 

Referring to FIG. 3, in step 52, the vertical interpolator 12 computes an estimated value, l v (h,v,t), for MP based on 
the pixel values at locations in / ODDf which are located in the horizontal lines above and below and in the same vertical 
line as MP or at l(h,v-1,t) and l(h,v+1,t), respectively, as shown in FIG. 4A, according to the following equation: 

45 

I v [h,v,t) - i(X (h.V'l. c) * r(h,vl,t>). (2) 

The processor 22 then stores the value of I v (h,v,t) in the memory 24. For purposes of simplicity in explaining the present 
invention, it is assumed that the processor 22 is pre-programmed for automatically storing in the memory 24 the results 

50 of computations for estimating a missing pixel value as well as any other computational results which are necessary 
for performing the operations described in the steps of the process 50. 

It is well known that performing an interpolation for estimating a missing pixel value from other known pixel values 
in the same frame or from other frames, or from approximations of missing pixel values in the same frame or other 
frames, is subject to error. The error associated with performing an interpolation estimation is typically computed by 

55 utilizing pixel values which are located in the neighborhood of or surrounding the pixels which were used in a particular 
interpolation estimation. For example, pixel values within 2 vertical lines of the pixels used in the vertical interpolation 
computation may be used to compute the vertical interpolation error. As more fully discussed below, the error associated 
with a particular interpolation is used for determining how much weight should be given to the interpolation when more 
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than one interpolation estimation value is utilized for generating an approximation of a deinterlaced frame. 

Referring to FIG. 3, in step 54, the vertical interpolator 12 suitably computes a vertical interpolation error value, 
Ey, in accordance with the equation: 

Then, in step 56, the linear temporal interpolator 1 2 computes an estimated value, l^h^t), for MP by averaging known 
values of spatially corresponding pixels in the interlaced frames which, in time, immediately precede and follow the 
10 frame being deinterlaced. The value for l t (h,v,t) is determined, in general, from the equation: 



15 



20 



45 



50 



I t [h,v,t) - (h, c-l> » X(J>, v,fc.i)). 



(4) 



where, as illustrated in FIG. 4C, the known pixel values in the frames l E vEN t -i a "d 'even^i are utilized in l(h,v,t-1)ax\6 
l(h,v,t+1), respectively. In step 58, the linear temporal interpolator 14computes an error, E t , associated with performing 
a linear temporal interpolation estimation according to the equation: 

E t- £ |ru,v,t-i) - id, v,t.i)\ 



In step 60, the processor 22 computes a first level approximation for MP based on the results of the vertical and 
linear temporal interpolations which were performed in steps 52 through 58. As conventional in the art, appropriate 
weighting factors, k„ are computed for each estimated interpolation value which is used in combination with other 
interpolation estimations for computing approximations of missing pixel values that are used for generating a deinter- 
25 laced frame. The weighting factor for an interpolation is computed based on a comparison of its error relative to the 
errors associated with other interpolations which are combined in the approximation, such that interpolation estimations 
having smaller errors relative to other interpolation estimations are weighed more heavily in the approximajjon of miss- 
ing pixel values used for generating a deinterlaced frame. In particular, in step 60, the weighting factors k v an6 /c,for 
the vertical and linear temporal interpolations, respectively, are computed from equations: 



30 



35 and 
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E + E 

40 c v c f 



: /=i — r < 6B ) 



After the weighting factors /c^and k t are computed, in step 62, the processor 22 computes a first order approximation, 
l(h,v,t), for MP from the equation: 

tfav.t) = k v l v (h,v,t) + k t l t (h,v,t), ( 7) 

As shown in equation [7], the vertical and linear temporal interpolation estimations are weighted according to their 
relative errors. In practical application of the present inventive technique, all missing pixel values in a frame being 
deinterlaced are approximated in accordance with steps 52 through 62 of the process 50, such that a first order ap- 
proximation of a deinterlaced frame is effectively generated in step 62. 

In steps 64 to 66, the orientational interpolator 16 computes an estimate for MP based on the pixel values in the 
first order deinterlaced frame. First, as well known in the art, the orientational interpolator 16 computes the displace- 
ment, 5 h , due to a local edge orientation for the frame matrix location of MP For an interlaced frame which is comprised 
of luminance pixel values, it is well known that 5 h is equal to 0.5/tan(p (h v) ). In accordance with the present invention, 
the orientational interpolator 16 determines the angle $ (h v) associated' with the local edge orientation for MP using 
known pixel value and estimated missing pixels which are' approximated in the first order deinterlaced frame, / ODCH in 
55 other words, based on a frame which was deinterlaced using both linear temporal and vertical interpolation techniques. 
For ease of illustration, it is assumed that all missing pixel values in l ODDt have been approximated, such that a dein- 
terlaced frame l ODDf may be referenced at the time that an orientational interpolation computation is performed. By 
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way of example and as shown in FIG. 4B, the orientational interpolator 16 may determine that the local edge passes 
through MP oriented from (h-1,v+1,t) to (h+1,v-,t), such that the pixel values at these spatial orientations are used in 
computing an orientational interpolation estimation value for MR 

After p and subsequently 5 h are determined, the orientational interpolator 16 then computes an orientation inter- 
5 polation estimate, i Q (h,v,t), for MP using the equation: 

I 0 (h, v, t) - i(r (i>-6 A , v-l, t) ♦ / v+\ § t) ). (9) 

Then, in step 66, the orientational interpolator 16 computes an error value, E ot associated with the orientational inter- 
to polation estimation, l ot using approximated pixel values from the first order deinterlaced frame l ODDf according to the 
equation: 

£ 0 - £jf U-6 A , v-l, t> - /<i-6 h , ir.l, c)|. (10) 

is in step 68, the processor 22 computes vertical, linear temporal and orientational interpolations weighting fac- 
tors*" * c andJc 0 , respectively, from the following equations: 

J_ 

20 ' — Y r < ("A) 

25 _ £ t 

*e""l i T <11B> 



30 



35 



40 



and 



_1 



i — I — r* < llc > 



- 1 



In step 69, using these weighting factors, the processor 22 computes a second order approximation, l(h,v,t), for MP 
from the equation: 

I (/j, v, C) - Jc v I v (h, v t t) . £ t I t ih, v, c) ♦ lT 0 I o (h # v, t) . (12) 



In practical application, all missing pixel values in the frame being deinterlaced are approximated in steps 68 and 69 
in order that a second order approximation of the frame being deinterlaced is generated. 

In steps 70 through 76, the bidirectional motion compensation interpolator 18 computes an estimated value for 
MP using conventional motion compensation techniques which account for motion of an object over time. Motion com- 

45 pensation is performed bidirectionally to allowfor better estimation of missing pixel values in the case of scene changes, 
such as objects which are covered in one frame but are uncovered in a preceding or following frame. As in the orien- 
tational interpolation performed in steps 64 through 66, approximated missing pixel values along with known pixel 
values in the frame being interlaced are utilized for performing initial computations, in other words, computing motion 
vectors, required for performing a motion compensation interpolation estimation. 

50 The motion compensation interpolator 1 8 first estimates motion over time, and then interpolates along the direction 

of the motion. Motion estimation is typically performed by matching a block of pixels in one frame with a block of pixels 
in another frame using a given search window. In an alternative embodiment, motion may be estimated by using pixel- 
based motion estimation techniques. In the exemplary embodiment of the present invention, a motion vector is pref- 
erably estimated using smaller sized blocks and larger search areas in order to obtain very accurate estimates of the 

55 spatial motion of objects across an image over time, as represented in a sequence of video frames. The use of smaller 
blocks with large search areas for identifying a motion vector, of course, requires more processing time. The added 
complexity and time of processing, however, is inconsequential because deinterlacing in accordance with the present 
invention is preferably performed off-line, that is, not in real time. In the event that deinterlacing is performed for real 
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time video data transmissions and display using progressive scanning, the typical selection of larger block sizes and 
smaller search areas for identifying a motion vector may compromise the quality of the deinterlaced frame ultimately 
obtained, as viewed on a display. In the case where the estimate of a motion vector is not extremely accurate, the 
result of the motion compensation interpolation estimation is likely to have significant error and not to be given adequate 
weight in generating estimated missing pixel values for an approximation of a deinterlaced frame in accordance with 
the present invention. For ease of reference, (d hb ,d Vb ) and(d hf d v ) are the backward and forward motion or displacement 
vectors, respectively, which would be typically computed. 

The motion compensation interpolator 18 identifies motion vectors for performing bidirectional, or forward and 
backward motion, interpolations. Both forward and backward motion interpolation estimations are performed in order 
to provide for an accurate estimation of the movement of an object over time, as represented in pixels located at spatial 
coordinates of a matrix that follow the motion vector. By way of example, an object captured in an image may appear 
in one frame, which is the frame being deinterlaced, but be completely absent from a preceding or following frame. 

The bidirectional interpolations utilize missing pixel values in deinterlaced frames which have been approximated 
to various levels of accuracy in accordance with the process 50. In particular, the bidirectional interpolation provides 
for a two part computation of a motion compensated interpolation value for MR In the first part, pixel values of a 
deinterlaced frame which, in time, immediately precedes the frame being deinterlaced are used for computing a back- 
ward motion interpolation estimation value for MR As shown in FIG. 4D, the immediately preceding frame is the frame 
'*evenm' wnicn nas b ee n completely deinterlaced according to the process 50. For the second part of the interpolation, 
pixel values of a deinterlaced frame which, in time, immediately follows the frame being deinterlaced are used for 
computing a forward motion interpolation estimation value for MR The immediately following frame, as shown in FIG. 
4D, is the frame r «™w which has been deinterlaced to a second level of approximation according to the process 
50. By way of example and referring to FIG. 4D, backward and forward motion vectors may identify the known pixel 
at (77-2, v+2) in the preceding frame and the missing pixel value at (v-2,h+1) in the following frame as the pixels asso- 
ciated with motion of an object encoded in MP. In this manner, the present invention obtains a more accurate estimate 
of a missing pixel value based on pixel values in a frame which has been deinterlaced. As in the other interpolations 
described above, the forward and backward interpolation estimation values are appropriately weighted in combination 
as more fully discussed below 

Referring to FIG. 3, in step 70, once the forward and backward motion vectors are identified, the motion compen- 
sation interpolator 18 computes the errors, E mb and E mp associated with backward and forward motion compensation 
interpolation estimations, respectively, using the following equations: 

£ -»"i5j r,i ' V ' t> " r " U - d V v - d ».' t - 1 >| (16) 



and 



Then, in step 7.2, the motion compensation interpolator 18 computes the forward and backward weighting factors, k b 
and kp respectively, required for generating a motion compensation interpolation estimation value for MP, using the 
equations: 

1 



and 



k f=— — (19) 



The total error for the motion compensation interpolation is expressed by 

E m = k b E m b + k f E m f . 

In step 74, the motion compensation interpolator 18 computes an estimated motion compensation interpolation value, 



E m = k b E m b + k f E m f . (20) 
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l m (h, v, t), for MP using the equation: 



(21) 



Finally, in step 76, the processor 22 computes a final approximation, l*(h,v,t), for MP, using the results of all four pre- 
viously performed interpolations, according to the equation: 

l*(h,v,t) = k v l v (h,v,t) + k t l t (h,v,t) + 



where the weighting factors for all the combined interpolation estimations are computed in accordance with the gen- 
eralized weighting expression: 



where the subscript x of ^ is set equal to v } t, o, and m for computing the weighting factors k p k Q and k mi respectively, 
associated with the vertical, linear temporal, orientation and motion compensation interpolation estimations. 

In a further aspect of the present invention, frames which are represented using the interlaced scan format to 
represent the chrominance or coloring of images may be deinterlaced for use in progressively scanned format. As well 
known in the art, an interlaced frame which is comprised of chrominance pixels is essentially a luminance matrix which 
has been downsampied by two horizontally. In other words, a chrominance interlaced frame contains chrominance 
pixels on alternate vertical lines of the frame matrix. 

The process 50, as explained above, may be similarly performed for deinterlacing a chrominance interlaced frame 
by implementing the following changes for the orientational and motion compensation interpolations. First, for the 
orientational interpolation, the angle (3 which would be obtained for deinterlacing of a given pixel (h,v,t) in the interlaced 
luminance frame is utilized in the estimation, except that the horizontal displacement expression, 5 h , is set equal to . 
25/tan(i^ ^. The expression for 5 h is modified because the distance between two pixels in the chrominance interlaced 
frame equals twice the distance between two pixels in the luminance interlaced frame. For motion compensation in- 
terpolation, for a given pixel (h,v,t) t the pixel locations identified by the motion vectors computed for deinterlacing the 
luminance interlaced frame are utilized in the estimation, unless a motion vector points to a nonexistent or missing 
pixel location in an interlaced chrominance frame. If this circumstance occurs, the pixel value used for performing the 
motion compensation interpolation is the average of the two pixel values horizontally adjacent to the missing pixel 
location pointed to by the motion vector. 

It is to be understood that the embodiments and variations shown and described above are illustrative of the 
principles of this invention only and that various modifications may be implemented by those skilled in the art without 
departing from the scope and spirit of the invention. 



1. A method for converting a sequence of interlaced video frames to a sequence of video frames formatted for pro- 
gressive scanning comprising the steps of: 

performing at least one first interpolation using known pixel values in the interlaced sequence of frames for 
estimating missing pixel values in an interlaced frame being deinterlaced; 

computing a first level approximation of missing pixel values in the interlaced frame being deinterlaced using 
the estimated missing pixel values obtained from the at least one first interpolations, wherein the at least one 
first interpolations are weighted in combination relative to the errors associated with each first interpolation in 
computing the first level deinterlaced frame; 

performing at least one second interpolation for estimating missing pixel values in the interlaced frame being 
deinterlaced using known pixel values in the sequence of interlaced frames and approximated missing pixel 
values in the first level deinterlaced frame; and, 

computing a second level approximation of missing pixel values in the interlaced frame being deinterlaced 
using the estimated missing pixel values obtained from the at least one first and second interpolations, wherein 
the at least one first and second interpolations are weighted in combination relative to the errors associated 



k 0 l 0 (h f v 9 t)^kj m (h g v 9 t) t 



(22) 



1 




(23) 



Claims 
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with each first and second interpolation which was performed. 
The method of claim 1 , further comprising the step of: 

performing subsequent at least one interpolations in a sequence for estimating missing pixel values using 
previously approximated missing pixel values in deinterlaced frames, said sequence comprised of interpola- 
tions which require a less accurate approximation of the missing pixel values for performing the respective 
interpolation estimation, said less accurate approximation interpolations being performed prior to the interpo- 
lations included in the sequence which require a more accurate approximation of the missing pixel values for 
performing the respective interpolations, 

and wherein the interpolation estimations are weighted in combination at each level of approximation of missing 
pixel values for the frame being deinterlaced. 

A method for converting a sequence of interlaced video frames to a sequence of video frames formatted for pro- 
gressive scanning, the method comprising the steps of: 

performing a vertical interpolation for estimating missing pixel values in an interlaced frame being deinterlaced 
using known pixel values in the interlaced frame being deinterlaced; 

performing a linear temporal interpolation for estimating missing pixel values in an interlaced frame being 
deinterlaced using known pixel values in the interlaced frames which, in time, immediately precede and follow 
the frame being deinterlaced; 

computing a first level approximation of missing pixel values in the interlaced frame being deinterlaced from 
a weighted combination of the estimated missing pixel values obtained from the vertical and linear temporal 
approximations; 

performing an orientational interpolation for estimating missing pixel values in the interlaced frame being dein- 
terlaced using known pixel values in the interlaced frame being deinterlaced and missing pixel values computed 
for the first level approximation of the deinterlaced frame; 

computing a second level approximation for missing pixel values in the interlaced frame being deinterlaced 
from a weighted combination of the estimated missing pixel values obtained from the vertical, linear temporal 
and orientational interpolations; 

performing bidirectional motion compensation interpolations for estimating missing pixel values in the inter- 
laced frame being deinterlaced using known and approximated missing pixel values in deinterlaced frames 
which, in time, immediately precede and follow the interlaced frame being deinterlaced, said preceding dein- 
terlaced frame comprising approximated missing pixel values obtained from the performance of vertical, linear 
temporal, orientational and bidirectional motion compensation interpolations whose results are weighted in 
combination, said following deinterlaced frame comprising approximated missing pixel values obtained from 
the performance of vertical, linear temporal and orientational interpolations whose results are weighted in 
combination; and, 

computing a final approximation of missing pixel values for the interlaced frame being deinterlaced using the 
estimated missing pixel values obtained from a weighted combination of the estimations obtained from the 
vertical, linear temporal, orientational and bidirectional motion compensation interpolations which were previ- 
ously performed with respect to the interlaced frame being deinterlaced. 

An apparatus for converting a sequence of interlaced video frames to a sequence of video frames formatted for 
progressive scanning, the apparatus comprising: 

a vertical interpolator for estimating missing pixel values in an interlaced frame being deinterlaced using known 
pixel values in the interlaced frame being deinterlaced; 

a linear temporal interpolator for estimating missing pixel values in an interlaced frame being deinterlaced 
using known pixel values in the interlaced frames which, in time, immediately precede and follow the frame 
being deinterlaced; 

a processor for computing a first level approximation of missing pixel values in the interlaced frame being 
deinterlaced from a weighted combination of the estimated missing pixel values obtained from the vertical and 
linear temporal approximations; 

an orientational interpolator for estimating missing pixel values in the interlaced frame being deinterlaced using 
known pixel values in the interlaced frame being deinterlaced and missing pixel values computed for the first 
level approximation of the deinterlaced frame, 

said processor computing a second level approximation for missing pixel values in the interlaced frame being 
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deinterlaced from a weighted combination of the estimated missing pixel values obtained from the vertical, 
linear temporal and orientational interpolations; and, 

a bidirectional motion compensation interpolator for estimating missing pixel values in the interlaced frame 
being deinterlaced using known and approximated missing pixel values in deinterlaced frames which, in time, 
immediately precede and follow the interlaced frame being deinterlaced, said preceding deinterlaced frame 
comprising approximated missing pixel values obtained from the performance of vertical, linear temporal, ori- 
entational and bidirectional motion compensation interpolations whose results are weighted in combination, 
said following deinterlaced frame comprising approximated missing pixel values obtained from the perform- 
ance of vertical, linear temporal, and orientational interpolations whose results are weighted in combination, 
said processor computing a final approximation of missing pixel values for the interlaced frame being deinter- 
laced using the estimated missing pixel values obtained from a weighted combination of the estimations ob- 
tained from the vertical, linear temporal, orientational and bidirectional motion compensation interpolations 
which were previously performed with respect to the interlaced frame being deinterlaced. 
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